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ABSTRACT 



The planning, design, development and field testing of 
an Instructional system for the training of educational 
disseminators/evaluators who are named Educational 
Information Consultants Is reported. A modular training 
program is described covering the acquisition of kr, owl edge, 
skills, and attitudes required to perform effectively 
In disseminating educational Information In various 
settings. Training materials and course sequences are 
focused on five processes: negotiation of a client's 

problem, retrieval of Research and Development information, 
transformation of retrieved information, communication of 
the above processes, and assessment of effectiveness. 

Course and performance objectives are presented. This 
thirty-class-hours training system has been developed 
and field tested as a self-contained, performance- 
oriented course containing lesson plans, exercises, 
readings, media, tests and instructor Inputs. Three 
field tests of the course are described. Results of 
pretest-posttest comparisons for each of the above processes 
as well as for the overall course indicate the effectiveness 
of this training program in educational settings and 
information centers in preparing educational dissemination 
personnel . 
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I. INTRODUCTION 



A. STATEMENT OF THE PROBLEM 

During the past decade, major strides have been made in creating 
educational information services and in bringing these services to the 
attention of a large and growing user population. The U.S. Office of 
Education has played a major role in these developments, particularly 
through the instrumentality of the ERIC data storage system. The growth 
of the ERIC Clearinghouses and of the various state and local educational 
information networks has sharpened awareness in the national educational 
community of the value of current and comprehensive information services. 
This in turn has increased the demand for such services. More and more 
educators are aware of the need to be appraised of noteworthy and tested 
practices throughout the country, in order to develop more broadly based 
programs of educational planning, implementation, and evaluation. 

That there is a corresponding need for people skilled in the various 
service functions which information networks perform is almost self- 
evident. As Havelock (1971) has pointed out, ". . . what we do not have 
and what we need so badly is a network of regional centres which can 
serve as truly comprehensive resource centres and resource linking 
centres with the skills and staff to be an effective mediating mechanism 
between R & D on the one hand and operating school districts on the 
other. " 

Increasing the number of centers, resources, and persons working 
effectively in them makes possible better utilization of current know- 
ledge and research about innovations and fresh attacks on old problems. 



Viable training programs for persons in such specialized tasks have not 
been readily available, however. 



B. RATIONALE FOR THE PROJECT 

Although curricula and educational practices in public schools have 
noticeably improved in the last decade, it is also clear that many of 
the innovative developments and practices have not been widely adopted. 
The reason for this is complex: 

1. The dynamics of the educational process (like the society which 
it mirrors) now produce problems of such intensity and 
frequency that needs for accurate, concise, and relevant 
information must be met. 

2. Much of the information about new educational developments does 
not exist in forms which are easily retrievable, intelligible, 
and applicable for school personnel. 

3. Schools do not now provide skills and arrangements required to 
apply and to make fully effective use of existing resources 
and information. 

4. There is only a partially effective information network among 
persons functioning in diffusion/dissemination roles and there 
are inadequate numbers of personnel trained for such roles. 

In 1941, Mort and Cornell found that it took 50 years for a 
practical educational innovation, kindergarten, to become widely adopted. 
Since that time, while that specific figure has been questioned (Brickell 
1961, Knapp 1959), no one has yet disputed the fact that it takes too 
long for sound educational innovations to become widespread in the 
schools. 

Educational investigators have studied the phenomenon, in the hopes 
of explaining it and thereby suggesting ways to alleviate it. The works 
of Carlson (1955), Miles (1964), Rogers (1962), Havelock (1969, 1967), 
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and Guba (1965) are most relevant. They point out that the problem of 
diffusion of educational innovations is also a communication problem, a 
matter for communication researchers and information scientists. 

Carlson postulates three barriers to change in the education system: 
(a) the absence of a change agent in the school structure, (b) a weak 
knowledge base about educational innovations, and (c) the "domestication" 
of the public school, meaning lack of accountability. Rogers postulates, 
as a result of his synthesis of diffusion of innovation in a variety of 
fields, that there are five stages in the adoption process: awareness, 
interest, evaluation, trial, and finally adoption. This scheme implies 
that information about the innovation at various levels of detail is a 
sine qua non to move users towards the final adoption decision. Rogers 
calls it the "communicability" of the innovation. Havelock (1967, 1969) 
makes the case that in order to speed the process of diffusion of inno- 
vations, a "linker," charged with the responsibility of bridging the gap 
between research and practice, must be instituted. This seems to be what 
Carlson refers to as a change agent. Havelock (1967, 1969), Guba 
(1965), and Farr (1969) reject the notion that the schools have the 
ability to assemble and use information directly from research. 

According to these investigators, the role of the "linker" is one of 
". . . gathering, processing, and distribution of educational knowledga,," 
Guba states that "some agency must be concerned with communicating the 
results of development activity back to the practitioner." A recent 
survey sponsored by USOE (National Center for Educational Research and 

p 

Development, 1969) reported that ". . . generally [the school admini- 
strator] is in no position to dig deeply into the literature, so ... he 
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needs to have a great deal of work done for him in the preparation of 
information." They point out the "... need for a kind of high-level 
journalism which brings the most significant facets of the research 
effort into non-techni cal form, both informative and attractive." Former 
Commissioner of Education James Allen (1969) also pointed out that "our 
first goal must be to get the good, new ideas and practices into use 
. . . and get them there quickly . . . [in the past, much of what] we 
have laboriously learned about educational theory and practice has been, 
to say the least, under-advertised, poorly packaged and thinly 
distributed." 

A number of studies and surveys conducted by the Communi cation Pro- 
gram of the Far West Laboratory have focused on the nature and extent of 
these needs. For example. Communication and utilization study (Lockheed, 
1967) confirmed that many school personnel are skeptical of the present 
efforts of educational research and development (R and D). The Communi- 
cation Program's experience with testing prototypes of information 
systems substantiates the belief that educational personnel do not have 
adequate information about new developments. Surveys (Hood and Hayes, 
1967; Chorness, Rittenhouse, and Heald, 1969a, 1969b; York, 1969), staff 
studies (Coney, Plaskett, Roggenbuck, and Hood, 1968; Mosher, 1969; 
Carlisle, 1968), and field testing of school planning and evaluation 
activities (Campbell *nd Markle, 1968; Hood, Carlisle, and Whitney, 1970) 
conducted by the Communication Program support the conclusion that most 
school organizations and personnel are poorly prepared to perceive, # 
process, evaluate, and utilize R and D Information. 



C. STATEMENT OF PURPOSE 



Responding to the nood briefly stated above, the U.S. Office of 
Education, Research Training Branch, funded the proposal of the Far Nest 
Laboratory for Educational Research and Development for the project 
reported In this paper. This project proposed to design and develop an 
Instructional system for the training of educational disseminators/- 
evaluators. The progra m Includes both prescribed content and Instruc- 
tional aids and accessories In a coordinated. Integrated Instructional 
system. Components of this system were Identified as objectives, 
learning tasks and activities. Instructional material and Instructors, 
media, and evaluation devices. 

The overall goal of the project was to develop training for 
personnel who would help to speed and make more effective the linkage 
between resources and school users of educational information. Such 
personnel would be trained to provide linkage services and would be able 
to staff effectively positions In the expanding educational resource 
center network. The objective of the training, then. Is to prepare 
personnel competent In Information linking skills. Including: (a) Identl 
fl cation of user needs, (b) retrieval of Information on educational 
products and programs related to specific user needs, (c) dissemination 
of such Information to the user/practitioner, and (d) assistance to the 
practitioner In making effective use of the Information. 

A secondary objective of the project was to prepare a training 

*The tame "Instructional system," "training program," "program," and 
"training" are used Interchangeably throughout this report. 
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system In which the primary Instructional node was active student 
participation In problem-solving tasks and activities, supplemented by 
Instructor management and appropriate written technical Information and 
guideline material. The system was to be In a sufficiently flexible form 
for application In a variety of training contexts and environments. 

D. TARGET AUOIENCE 

Development of a training program to achieve the above project goals 
assumes that there Is at least one Individual In each educational agency 
who needs the knowledge, attitudes, and skills to make effective use of 
outside Informational sources. This Individual needs to be able to 
convey, with considerable expertise, relevant and useful information to 
others In his agency. 

A prospective trainee, however, should meet some minimal qualifica- 
tions. He should be professionally mature to the extent that colleagues 
see him as "capable" in at least one role, e.g. , a supervisor of some 
Instructional service or department, the principal of a building, an 
agency-wide consultant for a subject or Instructional area, or the 
administrator of an Instructional program of some scope. He should be 
Involved, In either a current or a future position, in educational infor- 
mation dissemination or utilization activities at a school building, 
district office, county* regional, or state level. Persons meeting this 
qualification might, for example. Include a school principal or teacher 
specially assigned to Information linkage activities, an Information 
services specialist or assistant, a research analyst, a librarian, infor- 
mation specialist, or subject area consultant. 



It Is desirable, perhaps necessary, that the trainee have confidence 
and competence as an educator, but not necessarily expertise In some 
special or narrow field. Rather than career commitment to a particular 
subject matter area, a "generalist" viewpoint would be more appropriate. 
Any administrative and knowledge utilization skills would be added 
strengths. 

State, regional, county, or local educational agencies are sources 
of such personnel . 



la * r . 



o 

ERIC 



7 



II. DESCRIPTION OF THE INSTRUCTIONAL SYSTEM 



A. CONCEPTUAL BASIS 

The instructional system is based on a conceptual model developed 
prior to design of any of the course materials. (See Figure 1, p. 9, 

"A Process Model of the Educational Information Consultant Role.") The 
focal point of the model is the Linkage System, a network of local, 
district, county, regional, and/or state educational information 
dissemination services, which connects sources of educational research 
and development (R and D) information with school personnel responsible 
for classroom practice. These sources of information are labelled the 
"Resource System" and the school personnel, the "User System." The agent 
who provides services at various levels within the Linkage System is the 
Educational Information Consultant or EIC. The EIC interacts with both 
Resource and User Systems. 

In the model, both the Linkage System and the EIC are depicted as 
"responsive" to the User System — that Is, linkage services are activated 
primarily by requests from clients needing assistance. Response to a 
client consists of a sequence of activities which the EIC performs. The 
EIC begins Interaction with the User System when a client, seeking 
information to solve a problem, first contacts the EIC. The EIC, whether 
functioning at a local, district, county, regional, or state level, 
"responds" by helping the client to analyze, assess, and define specifi- 
cally the problem and corresponding information need. This process is 
labelled Negotiation (see Figure 1, Step I In the Linkage System). 

The next step. Retrieval , Is a process through which the EIC 



A PROCESS MODEl OF THE 
EOUCATIOMAL INFORMATION CONSULTANT ROLE 
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interacts with the Resource System. After planning a strategy for 
locating information relevant to the client's problem, the EIC searches 
selected sources. Once pertinent information is retrieved from these 
sources, the EIC then moves Into Step III, Transformation . This process 
is internal to the Linkage System. It requires that the EIC screen, 
organize, analyze, and/or synthesize the information retrieved until it 
is in a form which is "actionable." "Actionable" means that the infor- 
mation is compiled in a format and style appropriate for delivery to the 
client so that he can then use the information, with minimum effort, to 
solve his problem. 

At Step IV, Communication , the EIC again interfaces with the User 
System. The EIC presents to the client a package of transformed infor- 
mation on the problem. Whether verbal or written, communication is the 
process which makes clear to the client the results of the EIC's search. 
Communication completes the linkage of Resource and User Systems. 

Evaluation , Reformulation , and Adjustment , as presented in the model, 
are ongoing processes. Evaluation measures effectiveness on three 
dimensions: (1) the individual EIC's performance of each of the other 
four linkage processes; (2) the overall performance of the EIC role; and 
(3) the performance of the system setting in which the EIC functions. 
Evaluation is conducted to determine whether the Linkage System is 
successful in servicing the User System, in utilizing the Resource System, 
and in fulfilling the linkage functions. Reformulation and Adjustment 
introduce changes , implied by the findings of Evaluation, in linkage 
processes, functions, and modes of interaction with Resource and User 
Systems. 



B. ORGANIZATION AND FORMAT 

The instructional system developed to conform with this model is a 
thirty-hour training course entitled, "The Educational Information 
Consultant: Skills in Disseminating Educational Information." The course 
is organized in modules, ranging in length from three to six instruc- 
tional hours. Each module builds, for its theme, on one of the five major 
processes of the EIC role. A definition of each process, as well as the 
order for and amount of instructional time allotted to the module, 
follows: 



MODULE: 1 PROCESS: Negotiation INSTRUCTIONAL TIME: 20r = 

DEFINITION: 

To identify, analyze, and define specifically the problem and attendant 
information needs of a client. 

MODULE: 2 PROCESS: Retrieval INSTRUCTIONAL TIME: 20% 



DEFINITION: 

To develop a search strategy and locate, identify, and secure R and D 

information pertinent to the client*s problem and request. 

MODULE: 3 PROCESS: Transformation INSTRUCTIONAL TIME: 20% 

DEFINITION: : 

To screen, analyze and/or synthesize, and organize the results of the 
search in a form app r opriate for delivery to the cl ie nt. _ 

MODULE: 4 PROCESS: Communication INSTRUCTIONAL TIME: 15 t ,T == 

DEFINITION: 

To display and convey the results of the search to the client in a style 
appropriate for his use in finding a solution (si to the problem. 
l&bULE: 5 PROCESS: Evaluation INSTRUCTIONAL TIME: 10%* 



DEFINITION: ; : 

To assess the performance of the major EIC processes and overall role 
and the operational effectiveness of the setting within the linkage 
system; reformulate based on evaluation and follow-up evidence; and make 
adjustments in processes and functions. 

*The course also Includes an introductory segment on the emerging role 
and functional contexts of the EIC and a concluding segment for trainee 
testing and feedback. The introduction accounts for another 10% of the 
instructional time and a conclusion accounts for the final 10%. 



Actual instructional time totals 30 hours, including provisions for 
testing. This time is divided into 10 three-hour sessions which are 
suitable for daily or weekly class meetings. The meetings are spaced to 
permit the approximately two to four additional hours cf outside 
individual preparation needed for each session. A course outline, 
listing the outside assignments, is provided in the copy of the trainee 
manual which accompanies this report. 

C. INSTRUCTIONAL OBJECTIVES 

A set of instructional objectives was developed to provide both 
instructor and trainees with a description of the types of knowledge, 
skills, and attitudes which can be acquired as a result of learning 
experiences in the course. These instructional objectives specify 
intended outcomes for trainees who successfully complete the course — 
that is, they state v/hat trainees will be able to do st the end of 
instruction. There are three categories of instructional objectives: 

(1) knowledge, (2) affective, and (3) skill. The objectives in each 
category are identified in the sections below. 

In addition, a set of related performance objectives was derived to 
permit precise measurement of the degree to which trainees achieve the 
instructional objectives. Because the list of performance objectives is 
too lengthy to include in its entirety, however, only illustrative 
samples are provided in the sections which follow. 

1 . Knowledge Objectives 

By the end of the course, a trainee who has acquired the knowledge 

base essential to effective performance in the role of EIC will be 



able to 



a. Summarize research and theory on the interrelationship 
among educational resource, user, and linkage systems, 
particularly the dissemination mechanisms and networks 
which comprise such a linkage system. 

b. Define the basic components of the EIC role. 

c. Describe alternative functional contexts of the EIC role. 

d. Compare the functional characteristics of libraries and 
information centers. 

The following performance objective describes a behavior which, 

when demonstrated, would be one indicator of achievement of 

instructional objective (b) above: 

Given a list of 17 tasks, the trainee will choose the 
appropriate letter to indicate whether each task should (A) 
definitely, (B) only rarely, or (C) definitely not be performed 
by an EIC or his staff and will show a gain in total score 
correct between pretest and posttest. 

2. Affective Objectives 

Positive attitudinal response to the concept of, need for, and 
potential impact of the EIC role is a major goal of training. 
Accordingly, the instructional system is designed to encourage 
development of an affective state which is supportive of EIC 
functions and processes. By the end of the course, therefore, 
attitudes should be developed to the extent that the trainee 
will: 

a. Accept the usefulness of the educational information 
consultant role as defined, at least to the point of 
being willing to try it. 

b. Express commitment to the importance of helping 
schools improve opportunities for students to learn 
through use of well-tested research and development 
(R and D) information and products. 
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c. Express willingness to strive toward higher levels 
of performance in the knowledge and skills required 
to function effectively as an educational infor- 
mation consultant. 

d. Accept the potential of the dissemination function 
for improving school operations. 

Examples of related performance objectives which would indicate 

achievement of instructional objective (a) above include: 

(i) On a five-point scale, with "1" being "not at all 
useful" and "5" being "very useful," the trainee 
will rate "4" or better when asked to indicate 

how personally useful is the subject matter covered 
in the course. 

(ii) On a five-point scale, with "1" being definitely 
not useful and "5" being definitely useful, the 
trainee will rate "4" or better when asked to 
indicate whether the skills learned in the course 
are useful in his present or future position. 



3. Skill Objectives 

The ability to perform effectively and readily as an EIC 

is contingent on proficiency in the five major processes of 

negotiation, retrieval, transformation, communication, and 

evaluation. This proficiency, in turn, is based on ability 

to perform skills associated with and integral to each process. 

By the end of the course, a trainee who is acceptably proficient 

in a major process should demonstrate aptitude in each of the 

skills associated with that process at a specified level of 

competence. There are four different levels of competence: 

Orientati on — familiarity with the nature of the skill and 
its relationship to the major process. • 

Basic Skill — familiarity with the nature of the skill, as 
well as some ability to perform the skill, based on 
limited practice in applying i t. 
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Competency— abi 1 i ty to perform the skill effectively, 
based on repeated applications of the skill in varying 
situations. 

Expertise — competence in performing the skill and in 
teaching it to others. 

Skill objectives and corresponding levels of competence are listed 
below according to major process. 



Skill Objectives for the Negotiation Process 



Level of 
Competence* 



1) Ask questions about the client's problem which C 

elicit information essential to a precise formu- 
lation of the search request. 

2) Guide interviews in order to: (a) interpret and C 

clarify the client's information needs and (b) set 
priorities among them. 

3) Listen in order to comprehend fully and objectively C 
what is said in formal and informal interchange with 
client(s). 

4) Communicate orally in order to develop rapport with S 
the client. 

5) Conduct a client analysis, which includes making S 

inferences as to the client's concerns, motivations, 
and level of expertise. 

6) Compose precise written and oral descriptions of C 

the client's problem. 

7) Formulate a satisfactory contract with the client C 
for the service (s) to be provided. 

8) Recognize whether the statement of the client's C 

problem requires further clarification, analysis, 

or redefinition. 



*0 = Orientation 
S = Basi c Ski 1 1 
C = Competency 
E = Expertise 
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Level of 
Competence 



9) Question, discuss, and secure information from the S 
client when there is a need to redefine or restate 
the problem. 

The following are statements of performance objectives related 
to negotiation. The number(s) in parentheses after each performance 
objective identifies the related instructional objective. 

(i) As he listens to an eleven-minute tapescript of a simulated 
client-EIC interview to establish the parameters of a 
request for information, the trainee records the essential 
data for completing a search. A model report is provided 
by the program developers for the instructor's use in 
judging the adequacy of the student's reports. (6) 

(ii) After listening to the simulation and recording the 
essential data, the trainee will identify the types of 
questions used by the EIC, describe the errors in EIC 
response to the client's statements, and summarize the 
agreement made in the interview. Model answers are 
provided by the program developers for the instructor's 
use in judging the adequacy of the students' responses. 

(3) 

(111) Given a partially vague request for information in a 
client-EIC role-playing situation (e.g., for "topical 
materials for flexible use in an interest-based secondary 
school social studies course"), the student asks questions 
which will serve to reduce the ambiguity of the request 
and elicit definition of terms without engendering 
resentment or defensiveness on the part of the client, 
such as : 

(a) What are the learning goals which are to be 
particularly emphasized? 

(b) Are the materials to be for student use or for 
teacher reference? 

(c) By "flexible use," do you mean that the materials 
can be read at any time during the unit? 

(d) Can you tell me anything further about behaviors 
which the instruction should elicit? 
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At least one "learner-observer" rates the adequacy of the 
trainee's questions on an Observer Rating Form provided 
by the program developers. Rating on a 5-polnt scale 
will be obtained with the criterion for adequacy being 
that the trainee receives a rating of at least "adequate" 
on 70X of the Items. (1. 2, 4) 

(1v) Provided model (s) of negotiation/ Interview Intake forms 
and the assignment to devise his own Intake form, the 
student will design such a form of no more than two pages 
In length, which. In the judgment of the Instructor, Is 
adequate for recording all needed Information as judged 
against the model form provided by the program developers. 



Skin Objectives for the Retrieval Process 



Level of 
Competence 

1) Develop an efficient and comprehensive search c 

plan on the basis of: (a) Information obtained 

about the problem during negotiation with the 
client and (b) the characteristics of the 
resource system to be utilized. 

2) Conduct searches In various educational resource S 

systems and Information networks. 

3) Locate, select, and recover resources and products S 
relevant to the particular client's request. 

4) Simnarlze accurately the contents of a document S 

without reading It In detail. 

5) Keep records and establish and maintain filing 0 

systems. 

6) Recognize a need to modify or expand the search C 

plan and procedures In order to accommodate for 
constraints Imposed by the client, the E1C, or 
resource systems. 

The following are examples of related performance objectives for 
retrieval : 

CD Given a statement of a hypothetical client problem and 
asked to list descriptors according to the ERIC Thesaurus 
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which can be used to guide a search for information, the 
trainee will list at least four from a list of acceptable 
terms provided for this problem to the Instructor. (1) 

Cii) Given a list of five major steps Involved In the process 
of retrieval, the trainee will match 13 out of 16 skills 
and activities with the appropriate major steps. (1, 2, 3) 

(111) Given several printouts of ERIC abstracts from the DIALOG 
Information storage system, the student will assign the 
correct degree of relevance as established by the program 
developers for six out of eight abstracts. (3) 

(1v) Given a list of eight abbreviations of information 
resources (such as ERIC and CIJE), the trainee will 
correctly match five out of eight acronyms with brief 
descriptions of their purposes or functions. (2) 



Skill Objectives for the Transformation Process 

Level of 
Competence 



1) Screen resources and products for specific S 

relevance to a client's request. 

2) Organize and display screened Information. C 

3) Abstract accurately the contents of a document based S 
on thorough examination. 

4) Assess self-competence to synthesize information 0 

for the client. 

5) Discriminate appropriate alternative formats and S 

styles for presenting screened Information. 



The following are examples of performance objectives related to 
the transformation process: 

(1) Given a statement of a hypothetical client's problem. 
Including its limitations, and a list of resources and 
documents available to the EIC, the trainee will, to the 
Instructor's satisfaction. Indicate the Items he selects 
as appropriate, describe his plan for organizing them, 
and state his rationale for doing both. A criterion list 
of acceptable answers will be supplied by the program 
developers. (2, 5) 
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(ii) After retrieving relevant information on his "learner- 
client's” problem, the trainee will transform the infor- 
mation into a comprehensive written report to his client 
which describes his rationale for the selection and 
organization of the information, as judged by the instruc- 
tor to be at least "adequate" on a scale ranging from 
"very adequate" to "very inadequate." (2, 4) 

(iii) Given a research report, the trainee will prepare a one- 
page indicative abstract that adequately represents its 
content according to the standards for abstracting. (3) 

(iv) Given a report on a completed research study or project, 
the trainee will prepare a one- or two-page informative 
abstract that adequately describes the problem studied, 
the approach used, and the major conclusions according to 
the standards for abstracting. (3) 



Skill Objectives for the Communication Process 
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Level of 
Competence 

1) Report orally and in writing transformed infor- C 

mation to the client. 

2) Convey to the client judgments about the quality S 

and appropriateness of information presented. 

3) Identify and delineate ways in which the client can S 
make effective use of the information presented. 

4) Attend to client responses in such a way that S 

communication techniques can be adjusted. 

5) Verify that Information provided satisfies the S 

request as negotiated with the client. 

6) Compile and disseminate topical information of 0 

general professional interest. 

The following are related performance objectives for the communica- 
tion process: 

(i) Given a problem statement and a transformed information 
package, the student will prepare a letter of transmittal 
to the client which, in the judgment of the instructor, 

1 ndi cates : 



24 > 



19 



p) a restatement of the problem; 

(2) the organization of the package; 

(3) limitations, cautions, and exclusions on 
materials in the package; 

(4) suggested course of action for the client in 
using package; 

(5) EIC's judgment of self-competence in problem 
area; and 

(6) EIC's desire for feedback from the client. 

O, 2, 3) 

(11) After viewing a "role-playing" situation in which an EIC 
is conveying to a client a transformed information 
package, the trainee will indicate his recognition of 
elements essential to the communication process by 
completing a checklist, on which he specifies: 

(1) the EIC's strong points in the communication 
process ; 

(2) the EIC's weak points in the communication 
process; and 

(3) suggestions for more effective methods the EIC 
could use in communicating the information 
package to the client. 

Each trainee's response will be judged adequate to the 
extent that it generally coincides with trainee group 
consensus. (1, 4) 

(ill) After preparing the transformed information package on his 
"learner-client's" problem, the trainee will present the 
package to the "client" and will, in the judgment of both 
observer(s) and "client," effectively communicate its 
contents, including: 

p) a restatement of the problem; 

(2) the organization of the package; 

(3) limitations, cautions, and exclusions on 
materials in the package; 

(4) suggested course of action for the client in 
using package; 

(5) EIC's judgment of self-competence in problem 
area; and 

(6) request for feedback from the client. (1, 2, 

3, 4, 5) 
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Skill Objectives for the Evaluation Process 



LeveH of 
Competence 



1) Analyze and assess one's own performance of the S 

processes of negotiation, retrieval, transfor- 
mation, and communication. 

2) Obtain feedback and follow-up evidence from client S 
as to the effectiveness and utility of the service 
provided. 

3) Assess the overall value and effect of the EIC's 0 

operations to provide a basis for improvement of 
services. 

4) Recognize that functions, roles, or administrative 0 
procedures in the EIC system may need to be 

adjusted or changed as the result of evaluation. 

The following are statements of related performance objectives: 

(i) The trainee will construct an evaluation form to be used 
by an EIC in objectively assessing his performance in 

the five EIC processes, including criteria for successfully 
estimating the total effect of his service, as judged by 
the instructor to be "adequate" against a model provided 
by the program developers. (2, 3; 

(ii) Given a copy of a final report to a client of retrieved 
and transformed information, the trainee will write a 
critical evaluation of: 

(1) definitiveness of the problem statement; 

(2) appropriateness and completeness of the retrieved 
i nformati on ; 

(3) clarity of the transformed information package; 
and 

(4) comprehensiveness of the letter of transmittal. 

A model for judging effective evaluation will be provided 
to the instructor by the program developers. (1) 

(iii) After giving the transformed information package to his 

"learner-client," the EIC will be judged effective in his 
performance if the client indicates that the package meets 
the specifications of the negotiated contract. (2) 



D. 




INSTRUCTIONAL ACTIVITIES 

Two major Instructional modes are applied: 

1 • Learni nq-Task-Centered Mode . This is in sharp contrast with 
the traditional teacher-actor, learner-audience, teacher- 
teachi ng-a-subject mode. In the learning-task-centered 
approach (Banathy, 1969), the learner's environment is arranged 
so that he can master the knowledge, skills, and attitudes 
required to perform learning tasks described in the objectives. 
The learner is involved actively and intensively as the 
actor on a learning stage. The teacher sets the stage am* 5$ 
■involved in facilitating learning. 

2. Functional Content Sequence (Smith, 1968). The sequence of 
instruction, insofar as it makes materials more meaningful, 
can greatly affect the motivation of students. The essential 
form of functional context sequencing, which has been tested 
in a number of research studies and been shown capable of 
reducing student failure (Brown, et al_. , 1959; Goffard, et al . , 
1960; Hitchcock, Mager and Whipple, 1958; Hood, 1967a, 

1967b; Shoemaker, 1960), is to: 

a. Provide a meaningful orientation to the entire job for 
which a student is being trained; 

b. Introduce and organize jobs so that the relevance of each 
to the whole job can be demonstrated to the student at 
the time the topic is introduced; 

c. Follow a whole- to-part sequence in teaching functions or 
procedures; and 

d. Programme Instruction for each student so that he learns 
a graded series of tasks, each new task requiring the 
student to master new knowledges and skills. 
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With the exception of written technical or supplemental information 
and guidelines. Instructional activity consists primarily of direct 
student participation in simulated and/or real-life problem solving. 
Training research has demonstrated repeatedly that skills training is 
effective only if the trainee is actively engaged, both individually and 
in groups, in working through a carefully planned sequence of activities 
and problems. Technical information, guidelines, and theoretical 
instruction are therefore introduced only when appropriate, according to 
the principles of "Functional Context" sequencing. 

Instructional activities in each major module of the course have 
three phases: (1) preparation, (2) learning, and (3) application. During 
the preparation phase, trainees are introduced briefly to the major 
process of the module and are exposed to a diagnostic exercise which 
probes their ability to perform skills associated with the process. This 
is followed by the learning phase, which is an intensive training 
experience consisting of content presentation, written and oral exercises, 
and/or simulation activities. The module concludes with the application 
phase, during which trainees are exposed to a problem to test their 
knowledge of and/or copability to perform the major process. 

Throughout the course, trainees learn and apply EIC process skills 
primarily through large- and small-group, as well as Individual, 
activities, including role-playing, problem-solving, simulation, and 
decision-making exercises. Some instructor presentation, averaging 
approximately 20 minutes per session is also included. 

The Instructional activities are designed to fit the ten-session 
pattern of a quarter-term schedule. They can be expanded, however, for 
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compatibility with the longer fifteen-session pattern of a semester 
schedule. A description of the instructional activities in each session 
follows. 

Session 1: Emerging Pole of the Educational Information Consultant . 

The session opens with a brief presentation on the growing need for 
dissemination of educational information, the development of a linkage 
system to bring information from resource to user, and the role of the 
Educational Information Consultant ( E IC ) in that linkage system. Trainees 
are oriented to the basic processes comprising the EIC role: negotiation, 
retrieval, transformation, communication, and evaluation. Trainees then 
read and discuss case histories of persons operating in EIC roles at four 
different levels within the information dissemination network. 

The session closes with a communication game, which is played in 
pairs. Initially, the communication between each pair is one-way, one 
member giving the other directions for putting a "T-puzzle" together. 

After a limited time, two-way communication is opened. Finally, a visual 
demonstration is permitted. Trainees engage in a concluding discussion 
of basic problems involved in communication and draw correlations between 
their experience with the game and the interactive processes associated 
with the EIC role. 

Session 2: Negotiation . The session opens with a preparatory 
diagnostic exercise in which a telephone negotiation between an EIC and 
a client is presented on audio-tape. Trainees design a form for recording 
the results of the interview and also record observations of the negotia- 
tion process. They discuss this introduction to the topic of negotiation 
within the context of their outside reading and the ideas considered in 
Session 1. 

Trainees then form triads to participate in a structured role-playing 
exercise on the negotiation process. One member of the triad plays the 
role of an EIC; another, the role of a client; and the third, an observer. 
The persons playing the roles of client and EIC conduct an interview to 
define the client's specific instructional problem and to identify his 
corresponding information needs. The person in the observer role evaluates 
their effectiveness in performing the negotiation. Each triad then 
discusses descriptors selected by the EIC and client to search for infor- 
mation relevant to the problem. They also exchange feedback on their 
performance. A brief, large-group discussion for comparison of descrip- 
tors and for analysis of negotiation skills and techniques follows. 

Session 3: Negotiation. After a discussion of the preparatory 
readings on the negotiation process and a brief introduction to the 
forthcoming exercise, trainees divide into teams of four. Members 
alternate in the roles of EIC, client, and two observers so that each has 
an opportunity to act as a client and negotiate his own prepared problem 
with another team member playing the role of EIC. (The person acting as 
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EIC will adopt his client's problem as his own topic for subsequent 
retrieval, transformation, and communication assignments during the 
course.) During negotiation, each EIC records data about his client's 
problem and information needs on a checklist he had designed as a 
preparatory assignment. After two of the prepared problems have been 
negotiated, the team pauses to evaluate them. This feedback process is 
repeated when the remaining two client problems have been negotiated. 

For the application exercise, trainees record their analysis of the 
same audio-taped negotiation interview which opened the module. 

Session 4: Retrieval . A short objective preparatory exercise opens 
the first session on the retrieval process. A review and discussion of 
readings on problem definition, selecting descriptors, and search 
strategy follow. 

Trainees next examine a "Search Procedure Form." They complete an 
individual written exercise in which each records a brief statement of 
his client's problem and the descriptors selected. Trainees then group 
for a round-table "brainstorming" session, during which each trainee 
presents his problem to the group members who can suggest descriptors 
and/or approaches to employ in his search. 

Each trainee is introduced to the Guide to Educational Resources and 
is instructed how to work with the "Search Procedure Form.'* Trainees 
also examine selected secondary information sources, such as CIJE. R.I.E, 
and Education Index as possible starting points for their search. The 
session concludes with pairs of EIC's and clients meeting briefly to 
renegotiate or reformulate their problem. 

Session 5: Retrieval . An exercise on abbreviations, completed as 
a preparatory assignment, is the vehicle for a large-group discussion 
of frame of reference and its effect on understanding of the client's 
problem and definition of the search. Trainees also consider the 
influence of risk-taking behavior, knowledge-state, and personality 
attributes on EIC selection and use of resources. 

Discussion is then directed to alternative educational information 
sources. Two filmstrip presentations on the ERIC and DIALOG research 
information system are followed by an introduction to the Far West 
Laboratory's curriculum information system, ALERT (Alternatives for 
Learning through Educational Research and Technology). Trainees also 
evaluate individual and group performance on a second preparatory 
assignment, "Making Relevance Judgments," in which they analyze the 
pertinence of abstracts on ERIC documents to two search topics. The 
exercise is designed to sharpen skills in selecting and screening docu- 
ments appropriate to a client's problem. The session concludes with a 
written application exercise on the retrieval process. 
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Session 6: Transformation . An individual, written preparatory 
exercise, simulating the process of selecting, organizing, and synthe- 
sizing a package of information for a client, opens this module. Trainees 
then read and participate in a large-group discussion of the 
Transformation Guidelines , which outlines skills and techniques involved 
in transforming retrieved information, and the Transformation Checklist , 
which lists various types of transformation formats. 

Trainees next survey selected examples of transformed information — 
bibliographies, handbooks, state-of-the-art reports, literature reviews, 
etc. — and identify them by type, purpose, and intended audience. The 
concluding activity requires that trainees individually plan transforma- 
tion of information on their client's problem. 

Session 7: Transformation/Communication . The session begins with 
a large-group discussion of abstracting as a method of transformation, 
based on preparatory readings and a review of Guidelines for Abstracting . 
Trainees consider two types of abstracts — indicative and informative — 
and compare the abstract they have written as a preparatory assignment 
with samples of both types. The module concludes with a written 
application exercise, which is similar in structure to the preparatory 
exerci se . 

The module on Communication is introduced through an individual 
preparatory exercise in which each trainee composes a letter conveying 
a package of transformed information to a hypothetical client. Trainees 
then form small groups to engage in review of a preselected information 
package, with the group objective of developing a role-playing skit of 
an ElC/client, face-to-face communication. Selected members of each 
group then role-play the skit for their own and another group, whose 
members evaluate the performance. A large-group discussion of the role- 
playing activity and of the communication process concludes the session. 

Session 8: Communication . Trainees briefly discuss the Gui del i nes 
for Conveying Information to Client . They then reconvene in the teams 
of four formed during the negotiation module to rotate playing EIC, client, 
and two observers. Each EIC communicates the findings of his search. 

He presents to his client a package of information which he has organized 
and abstracted to satisfy the client's request. Observers give feedback 
to each EIC; the team, as a whole, evaluates the effectiveness of all 
four ElC/client communications. Exemplary EIC reports are selected by 
the teams for presentation to the rest of the trainees. A large-group 
critique of the activity concludes the session. 

Session 9: Evaluation . As an individual application exercise in 
communication, each trainee submits a letter conveying his information 
package to his client. Trainees then participate in a large-group 
discussion of the evaluation process. They consider problems, sources, 
methods, and techniques for evaluating performance of: (a) each major 
EIC process; (b) the EIC role; and (c; the center, library, or system 
in which the EIC is functioning. Individual trainees make oral presen- 
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tations to the large group, evaluating information centers they have 
visited and utilized. Trainees also review and discuss case histories 
and readings on information centers and personnel in terms of the 
evaluation process. 

Session 10: Evaluation . The course concludes with a written final 
examination — an application exercise on the major EIC processes and skills 
covered in the five instructional modules. Trainees then discuss and 
evaluate the course and receive feedback on performance from the 
instructor. 

E. MATERIALS 

A looseleaf notebook contains the job aids, readings, learning 
exercises, guidelines, and other accessories developed for trainee use 
in the course. These materials in the trainee notebook are grouped into 
ten sections, which correspond to the ten-session format of the course. 

The instructor's notebook consists of the trainee materials for the ten 
sessions, accompanied by a schedule and detailed notes on the format and 
content for each three-hour instructional session. All trainee and 
instructor materials, including diagnostic exercises, are compiled in 
the sample notebook accompanying this report. 

Audiovisual materials now built into the instructional time include 
a filmstrip with tape on the ALERT information system, an audio-tape on 
the negotiation process, and a slide- tape set describing the ERIC DIALOG 
system. Copies of these materials were available in the Far West 
Laboratory. Two films — "The Paper Blizzard" (22 minutes) and "SPIRES 
and BALLOTS Report" (12 minutes) — were optional materials for use in 
the retrieval module. Neither film was available from the Far West 
Laboratory, but both were rented, at nominal cost, from film distributors. 
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III. DEVELOPMENT AND EVALUATION OF THE INSTRUCTIONAL SYSTEM 

The strategy for development of the instructional system took the 
following sequence: 

Design and testing of the conceptual model; 

Planning and design of the training program; 

Preliminary field testing and evaluation of the training 
program ; 

Main field test development and evaluation; and 

Preliminary operational field testing and analysis. 

The above sequence observes the product development strategy of the 
Far West Laboratory. That strategy unfolds in the following way: after 
conceptualization, planning, and a training plan have been completed, a 
preliminary field test Is conducted with a small, but representative, 
sample of the target audience. Following this test, a main form of the 
product is then developed. This form incorporates any revisions and 
changes needed to Insure effective performance of the product. The 
product then undergoes a main field test with a larger sample of the 
target audience. This test is conducted under Laboratory supervision and 
is designed to check the product's effectiveness in meeting stated 
objectives. During this field test stage, revised and refined evaluation 
procedures, formats, and instruments are checked. 

Generally, Laboratory products are carried through an operational 
field test stage, during which the final product is tested for its 
success in actual settings without Laboratory supervision. The present 
project was not funded for this phase; however, a feasibility test. 
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preparatory to the normal operational test, was conducted with a snail 
target audience sanple to determine whether modifications and revisions 
would be needed to make the main form of the product operational and 
transportable. (A proposal for additional funding to carry the develop- 
ment of the Instructional system through the operational testing and 
dissemination stages has been submitted to and approved by the U. S. 
Office of Education. National Center for Educational Communication.) 

The procedures and results of each of the five phases In the 
developmental strategy are reported In the following sections. 

A. DESIGN AND TESTING OF THE CONCEPTUAL HODEL 

The Initial activity In this phase of development Involved a survey 
of the literature. This produced some 500 abstracted Items for staff 
reference, furnished Input for the development of conceptual models, and 
permitted ready access to documents of greatest relevance to the project. 

Project staff also researched and prepared working papers which 
described the boundaries of the training system, defined training needs 
and requirements, specified resources available and constraints limiting 
the design and training effort, and stated specifications of expected 
performance. The staff also developed alternative conceptual models of 
the Instructional system. 

The working papers and models were presented to a panel of 
consultants at a two-day conference held In August 1970. Participants 
In that conference. In addition to project staff and Far Nest Laboratory 
advisers. Included six consultant-experts Involved In various levels of 
service In the educational Information dissemination/evaluation network. 



specifically. In local and county PACE centers, a regional educational 
resources center, an ERIC clearinghouse, and In a State educational 
program, planning, and development agency. 

The outcome of the conference was the selection of a conceptual 
• model for the Instructional system and the Identification of 
corresponding goals for training. The consultants described, as an 
additional outcome of the conference, behavioral task areas In which 
trainees should develop competence sufficient to perform effectively the 
major processes of a dissemination/evaluation role. 

Project staff subsequently defined and displayed the conceptual 
model and Identified five major processes as the focal areas for 
training: (1) negotiation of a client question, (2) retrieval of relevant 
Information from resources* (3) transformation of the retrieved Infor- 
mation Into a form actionable for the client, (4) communication of the 
Information, and (5) evaluation, reformulation, and adjustment. (See 
Chapter II.A., CONCEPTUAL BASIS, for a detailed explanation of the model.) 

With the list of goals and task areas suggested by the conferees 
as a basis, the project staff also developed a set of training goals for 
the Instructional system and a list of knowledge and skill competences 
for dissemination/evaluation personnel completing the training. 

These goals and competences, along with a summary report on the 
results of the conference, were then presented to all six consultants 
for their review and/or comment. Three of the consultants were asked to 
submit a critique of the goals and competences and to respond to a 
detailed questionnaire on the proposed training model and Instructional 




30 



37 



techniques. Their affirmative response constituted final validation of 
the model and proposed training goals and objectives for the Instruc- 
tional system. 

B. PLANNING AND DESIGN OF THE TRAINING PROGRAM 

The planning, design, and selection of training content and 
methodology constituted the next phase of development. An extensive 
search and thorough screening of available literature on educational 
Information dissemination, knowledge utilization, library and Information 
science, and processes for effecting educational change were conducted 
for four months by a project staff of three, with librarian assistance. 
Four ERIC searches by the DIALOG computerized retrieval system were 
completed. Several hundred bibliographical references were culled, 
categorized, and abstracted or retained as original documents. 

. Following exploration of training programs relevant to the model 
and also completion of site visits to operating educational Information 
centers. Instructional objectives were formulated and training content 
and method selected. A detailed course plan, outlining the time schedule, 
format, specific activities and content, and objectives for each of ten 
three-hour Instructional sessions, was prepared, along with a narrative 
description of the sessions. Prototype learning exercises were developed 
and validated Informally with representative samples of the target 
audience. 

A second conference was then held In December, 1970, with several 
of the consultants who participated In the first phase of development 
and with others who were to be subsequently Involved as Instructors and 



field test observers. This group selected and tried out learning 

( 

exercises and submitted written critiques of the couirse plan, student 
materials, and evaluation devices presented for their review. The 
conferees approved the overall plan and design of content and materials 
for the training. 

C. PRELIMINARY FIELD TESTING AND EVALUATION OF THE TRAINING PROGRAM 

The preliminary field test (PFT) of the instructional system was 
conducted: (a) to determine the feasibility of the training program 
design for meeting stated objectives; (b) to identify the strengths and 
weaknesses In the training activities; (c) to collect and assess data on 
the evaluation procedures and instruments; and (d) to provide a basis 
for revising elements as needed. 

1. Method 

The training program was tested in the form of a thirty-hour course 
with ten weekly sessions of three-hour duration. Since determination of 
the feasibility of this design was a major purpose of the PFT, the 
selection of an appropriate test site was given greater emphasis than 
direct selection of subjects. The University of California (U.C.), 
Berkeley, was therefore chosen because the course could be offered for 
three quarter- term units of credit through the U.C. Extension Program. 

Both the University and the Laboratory announced the course through 
limited direct mailings to and postings In Bay Area school districts, 
libraries, media and resource centers, and other educational agencies 
and organizations, such as the California Association of School Libraries. 

Eleven persons subsequently enrolled in the course, which was offered 
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during the Winter quarter from the beginning of January to the end of 
March 1971. This self-selected sample proved to be a mixed, but 
appropriate population. It included personnel in various information 
dissemination positions and roles, such as librarian for a federally 
funded community action project, director of a school district media 
center, chairman of a university library science department, school 
librarian and consultant, and research assistant in an educational 
development agency. Dr. Wayne E. Rosenoff, director of this project, 
served as Instructor for the course. Dr. Robert Peterson, director of 
the Instructional Planning and Management System Component of the 
Laboratory's Communication Program, served as Independent observer. 

Field test procedure stipulated that the course be conducted 
according to the schedule and format for instruction approved during 
the preceding phase of development. Eleven participant-trainees 
completed the ten sessions, during which they tried or used all 
preparatory, learning, and application exercises and materials. In 
addition, they completed experimental forms of the overall course and 
modular pretests and posttests. 

Participants were asked to judge the usefulness of the skills 
emphasized in the course and the quality of the instructional methods 
and materials used in the training. Throughout the course, participants 
were given the opportunity to respond freely to questions posed by the 
project staff, as well as to draw attention to problems and questions 
not previously identified by the staff. 

2. Results 

Participants indicated that there exists a need and desire for 
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training toward a role in educational information dissemination. They 
saw that the knowledge and skills gained from taking the course were 
useful, indicating as particularly valuable the acquisition of skills in 
the major processes of negotiation, retrieval, transformation, 
communication, and evaluation. 

That the participants learned the skills and knowledge was confirmed 
by their performance on an experimental paper-and-penci 1 cognitive 
exercise administered at the beginning and again at the end of the course. 

' Although the combination objective/subjective test was developed 

primarily to check the cognitive evaluation format and procedure, results 
showed that participants made statistically significant gains from 
pretest to posttest. When pretest/posttest comparisons of total mean 
scores were made as judged by two independent raters, significant results 
were obtained (t * 4.16, df = 7, p < .01 and t = 7.21 , df = 7, p < .01 ). 

Reactions to elements of the training varied. According to the 
observer's report, instructional activities and exercises which required 
active trainee participation seemed to have the most success. 

Organization of trainees into small working groups or teams seemed to 
promote active individual participation, as did role-playing exercises 
and simulations. Discussions and brainstorming exercises also 
demonstrated the value of informal exchange as an Instruction technique. 

It was suggested that some elements of the training which, as 
presented, appeared not useful to the course, be changed or else dropped. 
The Introductory oral presentation on the levels within the educational 
information network was one such example of an ineffective instructional 
format. Another Individual written exercise in classification was 
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assessed as too projective a technique for trainees to use in its existing 
form as a basis for in-class discussion. 

It was also suggested that the scheduling of activities was too 
rigid and that more flexibility and on-the-spot accommodation of trainees' 

. desires for discussion be built into the course. In addition, the need 
for development of a more precise set of instructor guides or notes, as 
well as a more effective mechanism or procedure for feedback on trainee 
performance, was indicated. 

Generally, however, the results of the PFT were favorable regarding 
the design of the course, its modular format, and its clear achievement 
of a very high level of active trainee participation. 

D. MAIN FIELD TEST DEVELOPMENT AND EVALUATION 

The main field test (MFT) represented a systematic effort to evaluate 
the training program and to provide information on its effectiveness in 
achieving the stated objectives. The main field test was also used to 
identify ways in which parts of the program might be improved. Following 
the MFT, decisions were made about possible modifications needed to 
correct any identified deficiencies in it. 

1. Method 

The MFT was conducted from mid-February through mid-May 1971 in 
cooperation with the Graduate School of Education, San Francisco State 
College (SFSC) with a sample of the target audience almost twice as large 
as that in the PFT. Again, the choice of test site took precedence over 
the selection of subjects. Scheduling the course at SFSC provided the 
opportunity to check the suitability of the training form for a semes ter- 
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term schedule. The course was offered for three semester units of credit 
through the Department of Interdisciplinary Studies. Of the fifteen 
sessions scheduled for the semester, ten were allotted intact for the 
implementation of the complete thirty-hour course, one for an introduction 
to the work of the Far West Laboratory and to the purpose of the MFT, one 
for a presentation on use of the college library facilities, one for a 
free question-and-answer session, one for feedback on and discussion of 
preliminary results of the MFT, and one for Independent study. 

The 18 persons who participated in the MFT were, for the most part, 
regularly enrolled graduate students in education, notably in the areas 
of interdisciplinary studies and educational technology. They had been 
recruited via posted announcements and word-of-mouth within the School 
of Education. They were, by nature and motivation, highly Interested in 
a new field. Many of the 18 held jobs while completing work toward the 
master's degree. Their occupations ranged from elementary or secondary 
school teacher. Teacher Corps Instructor, audiovisual equipment salesman, 
director of a curriculum project, and educational research analyst to 
research chemist and business training analyst. 

Instructor for the course was Freeman Elzey, Research Associate at 
SFSC. Nancy Adel son, a program assistant on the project staff, served 
as official Laboratory observer. Other members of the staff responsible 
for development of the instructional system occasionally observed the 
field test and were actively involved as coordinators of the field- 
testing activities. 

2. Procedure and Format 

Evidence was collected on three aspects of the instructional system: 
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(1) trainee cognitive growth; (2) trainee attitudinal or affective 
response; and (3) performance of all exercises and the course schedule. 
Both summative and formative evaluation were thus conducted, with 
emphasis on the latter. 

The procedure and format for measuring the overall cognitive effect 
involved the administration of a written exercise on the whole course at 
the beginning (Session 1) and again at the conclusion (Session 10) of 
training. Written exercises designed to measure the cognitive effects 
of individual modules were also administered on a pretest/ posttest basis 
in four of the five modules. 

Affective questionnaires were administered at the conclusion of the 
Negotiation module. Retrieval module, Transformation/Communication modules 
and Evaluation module and also at the conclusion of the course. These 
same questionnaires also contained product-rating items. 

In addition, transactional records on each session of the MFT were 
kept by the Laboratory observer to check the extent to which the course 
was being implemented as planned and also to collect evidence on the 
nature of transactions between students-and-instructor and students- 
and-students. 

3. MFT Instruments 

The overall course cognitive test (presented in Appendix A-l) 
included multiple-choice, matching, and completion objective items, as 
well as some short-answer subjective items. The modular tests were as 
follows: 
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Module 1: Negotiation Exercise Objective and short-answer items, 

requiring design of a recording form 
and brief responses to questions on 
arv audio-tape of an interview. 

Module 2: Retrieval Exercise Objective, matching, and completion 

items on phases of the retrieval 
process. 

Module 3: Transformation Exercise Open-ended subjective items, 

requiring review of the problem 
statements and design of information 
package. 

Module 4: Communication Exerci-a Subjective test, requiring composition 

of a letter of conveyance to client. 

Module 5: Evaluation No pretest/posttest. 

(See Appendix A-3 to A-6 for MFT cognitive evaluation forms.) 

The questionnaires containing affective scales asked trainees to 
make ratings of the extent to ( y/hi ch they intended to apply the skills 
and knowledge covered in training t ^ their real-life work. These same 
questionnaires also included items asking trainees to make ratings of 
the quality of the elements and methods used in the training and of each 
module as a whole. Open-ended questions were included in the whole 
course affective questionnaire to elicit free responses regarding the 
training program. (See Appendix B-l to B-5 for MFT combined affective 
and product evaluation forms.) 

4. MFT Form 

The MFT form of the instructional system included both trainee and 
instructor materials for ten instructional sessions. The trainee 
materials were contained in a looseleaf notebook and consisted of 
preparatory and in-class readings, exercises, guidelines, checklists, 
information packets, and a Guide to Educational Resources . These 



materials were distributed session-by-session. 

The instructor's materials consisted of the trainee notebook, with 
a schedule and set of instructor's notes for each session. The 
instructor's notes were detailed, providing specific timing, topics, 

. background data and/or information for instructor review, and specific 
explanations and directions to be given to trainees. 

In addition, each trainee was provided in Session 1 with a course 
outline, listing preparation assignments and topics for each of the ten 
sessions. 

5. Results 

Course Cognitive Outcomes . Cognitive evaluation procedures 
assessed the extent to which knowledge, processes, and skills required 
of the EIC were attained. Table 1 on p. 40 presents the results of the 
course pretest/posttest comparisons. (Appendix A-l contains the overall 
course exercise to which Table 1 data refers.) For statistical signifi- 
cance, a level of .05 or less was required for these and all other 
statistical tests reported. As shown in Table 1 , on the following page, 
there was a significant increase in mean scores from pretest to posttest 
as judged by the two Independent raters. The reliability of the judges' 
rating was acceptable. It was somewhat higher for the pretest ratings 

i 

(.87) than for the posttest ratings (.76). These results indicate that 
an overall cognitive change, in a positive direction, was effected by 
the instructional system and training. 

The statistical pretest/posttest comparisons for each of the 
objectively scored items are presented in Table 2, p. 41. There was a 
significant increase in the mean scores from pretest to posttest in 

. # 



39 



TABLE 1 

Results of the Pretest /Post test Comparisons of Combined Objectively 
and Subjectively Scored Items in the Overall Course Diagnostic Exercise for Site }/ 
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Results of the Pretest/Posttest Comparisons of Objectively Scored Items 

with Multiple Correct Answers in the Overall Course Diagnostic Exercise for Site #2 
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three out of the four questions. Thus, there was some Increase In the 
trainees' ability to specify differences and similarities between 
libraries and Information centers, to define task responsibilities of 
the EIC role, and to Identify commonly used acronyms for Information 
resources. 

Table 3, p. 43, presents the pretest-posttest comparisons for the 
objectively scored Items In the overall course exercise. As Indicated 
In this table, only two out of the five questions showed significant 
change. These questions dealt with matching Information resources with 
brief descriptions of their purpose. The Information tested In Problem 
III, questions 2-6, was subsequently deleted. 

Table 4, p. 44, presents the statistical comparisons, as rated by 
two j-lqes, fo*' each of the subjectively scored items on the overall 
course exercise. All Increases and mean scores for these Items for 
both raters were statistically significant, except for Rater W on 
"qi/^Mty of rationale" for questions 1-5. Rater W generally provided 
lower ratings than Rater N on both pretest and posttest, causing the 
percentage cf absolute agreement to be rather low, particularly on the 
posttest. Reliability of pretest/posttest ratings ranged from r = .12 
to r ■ .87. The ratings were generally higher on pretest than on 
posttest. 

Module Cognitive Outcomes . Tables C-l through C-4 In Appendix C 
show the statistical pretest/posttest comparisons of the cognitive 
exercises for the Negotiation, Retrieval, and Communication modules. 
(Unfortunately, the student response sheets for the Transformation 
module exercise were destroyed by fire prior to data analysis. There 
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Results of the Pretest/Posttest Comparisons of Objectively Scored Items 

with a Single Correct Answer In the Overall Course Diagnostic Exercise for Site #2 



















4J 








4J 4J 


9 








9 9 










VI VI 
© 4J 


£ 9 








* 8 










«** 








° 8 


v On 
U 








tJ *4 


8 8 








© o 


U 








u © 


O VI 








8 8 


o o 

58 








o o 








S3 


M 8 








© O 

•o 










11 


11 








1 


■ 








< 


O 








• • 


• H 








CO CO 


cn o 


m 




0. 


• • 


• o 


o 






S3 SB 


SB • 


• 








V 


V 




© 

© 


oo o 


o 


N 




M 


n oo 


in o 


«H 




3 


H 


M 


CO 


6 


cr 

co 




M 




Q 


co <r 


2 

13 






O H 


o o 


H 




55 


00 00 


00 oo 


00 






H H 


M M 


M 






•• CN| CO 


m 


SO 






M 

m a a 


§ 8 


0 






MOO 


O 






•H -H 


•H -H 


•H 






S Vi V> 


V» VI 


VI 






53 CD CO 

3 3 © 

PQ 3 3 


© © 


© 






3 3 


8 






a ^ °* 

tu 


O' O' 


O' 



5 

© 

4J 

10 

u 

M 
4i 

I 

u 

m 

. I 

11 

“1 

•S" 

sa 

3 k 

V) M 

$ 

• s 
© 

60 

J« 

° s 
8* 
8 . 
3 3 

«M O 

a S 

# o 

CO CO 



Vi 

O 

4H 



s & 

CD PQ 

*8 5 
S " 
5 8 

£ *M 




43 



So 



TABLE 4 

Results of the Pretest/Posttest Comparisons of Subjectively 
Scored Items in the Overall Course Diagnostic Exercise for Site 12 
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was no written exercise for the Evaluation Module.) In general, there 
were statistically significant Increases In knowledge and skills 
associated with the processes of negotiation, retrieval, and communica- 
tion. Trainees showed their capability to design negotiation Interview 
• forms and to Interpret the results of a simulated Interview, to Identify 
tasks essential to the planning and conduct of a search strategy, to 
select and describe a plan for transforming Information on a client's 
problem, and to prepare a comprehensive letter conveying the Information 
package to a client. 

Course Affective Outcomes. Affective outcomes were measured in 
terms of trainee attitudes toward the Instructional system or training 
as a whole, toward Individual elements, that Is, the Instructional 
modules, and toward subelements, that is, specific exercises and 
materials. The mlnlmimi standard for acceptability of ratings In the 
affective evaluation, both for the entire course and for Individual 
modules. Is that the mean rating for each Item be 4.5 (lower limit of 
"5" rating) on a seven-point scale or. In some ratings, 3.5 (lower limit 
of “4" rating) on a five point scale. 

Table 5, p. 46, presents trainee means on rated Items from the 
affective questionnaire on the entire course. (See Appendix B-l of 
this document.) The standard was achieved on 83% (or 10 out of 12) of 
the Items. Response to these Items indicated the extent to which 
trainees valued the course and their experience during training. The 
trainees apparently considered that the skills learned In the course 
will be useful on the job. On a 5-point scale, with "1" being definitely 
not useful and "5" being definitely useful, all but one subject 
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Results for Affective Questionnaire 
Site 02 

Sessions 1 - 10 



ITEM 


Achieved 

Mean SEM N Standard 


4. Do you consider that the job you now hold 
(or are working toward) would require 
you to use the skills you*ve learned in 
this course? 


4.56* 18 yes 


5. How personally useful do you feel the 
subject matter covered in this course 
will be to you? 


4.61* 18 *■*. s 


7. How did you enloy this course? 


4.22* 18 yes 


11. After reading each of the items below, 
t> lease circle the number which most 
nearly corresponds with your opinion. 

a. This course helped me to clarify my 
career plans. 

b. I would like to take this kind of 
course again. 

c. I am very satisfied with this course. 

d. I am now quite excited about the field 

of educational information dissemination. 

e. I would recommend that others take this 
course. 


3.12 17 no 

4.33 18 no 

5.24 17 yes 

4.70 20 yes 

6.12 17 yes 


12. For each of the following items, please 
circle the number which most nearly 
corresponds now with your feelings upon 
completing this course. 

a . The relevance of the training in this 
mump to vou in vour present position. 

b. The relevance of the training in this 
course to vou in your future position. 

c. The value of this training in comparison 
to other training you have received in 
college or university courses. 

d. Your interest in taking more advanced 
courses in this field. 


5.47 17 yes 
6.12 16 yes 
5.47 17 yes 

5.17 18 yes 



*These items scored on a five-point scale. All others were scored on a 



seven— point scale. 
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responded at "4" or "5." There were 67% who responded at "5" and 28%, 
at "4." These trainees also seemed to enjoy the course— 89% of them 
marked "4" or "5" on a 5-point scale varying from not at all ("1") to 
very much ("5") enjoyed the course. 

When asked to indicate on a 7-point scale, varying from definitely 
no ("1") to definitely yes ("7"), whether they would recommend that 
others take this course, 94% of the trainees checked "5" or better. 

When comparing the value of their training in the course to other 
training received in college or university courses, 76% of the trainees 
responded at 5 or better on a 7-point scale varying from "1", much less 
value, to "7", much more value. 

Module Affective Outcomes . Tables C-5 through C-8 in Appendix C 
present the trainees' ratings on affective items for individual modules 
on Negotiation, Retrieval, Transformation/Communication, and Evaluation. 
(These instruments are presented as Appendix items B-l through B-5.) 

The percentage of items achieving the standard for each module is shown 



below: 

Module Percentage 

Negotiation 67% 

Retrieval 71% 

Trans formati on/Communication 86% 

Evaluation 86% 



Summary . Reaction to participation in the training was, then, 
generally favorable. Several trainees, however, did indicate that the 
time schedule was overcrowded, considering the amount and nature of the 
activities to be covered. When asked what they did not like about the 
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course, trainees commented: "Too much to cover in a session"; "the 
time squeeze on activities"; "the frustration of not having enough time 

* 

for creative learning experiences." These comments are supported by the 
conclusions of the course instructor and observer, who Indicated that 
compressing the course Into 30 class hours did not allow sufficient 
time for trainees to discuss and thoroughly integrate what they were ' 
learning. 

Reaction to the experience of teaching a complete, "pre-packaged," 

student-oriented course can perhaps be derived from the following 

comments of the instructor: 

The amount of student activity built into this course 
(as opposed to the passivity of listening to lectures), 
the rapid sequencing of the course, the large amount 
of material provided the student, and the assignment 
of a project that involves serving another student, 
were indeed a departure from the usual academic 
process. There Is no question that after the initial 
"shock’ 1 of this process, the students adapted quite 
well and saw it as a very positive and rewarding 
learning experience. 

E. PRELIMINARY OPERATIONAL FIELD TEST 

A preliminary operational field test (POFT) was conducted: (a) to 

test the feasibility of the system as a "transportable" package, i.e., 

to test whether the first design and the content of the guideline v 

materials provided sufficient support for the* instructional manager to 

conduct the course successfully without active intervention of a 

development team member; (b) to test .the value of having the instruction 

in an information center; and (c) to test the value of t^e training for 

personnel already functioning a*t varioys levels in the EIC role, and at 

various stages of professional competence. 
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1 . Method 



The field test site was the San Mateo County Education Resources 
Center, Redwood City, California. Ten weekly sessions were held April 1 
through June 10, 1971. The Center Director, Dr. Frank Mattas, served as 
instructor, and Dr. Sanford Glovinsky served as the independent observer. 
Both had participated in the early planning conference and in the 
conference held just prior to the preliminary field test. 

The trainee population consisted of five members of the Education. 
Resources Center staff, a member of the San Mateo PACE Center staff, 
two district librarians, and a public information specialist from a 
nearby district. Both classified personnel and professional persons 
employed in the information field were represented in this class. 

The course was offered through the University of California, 
Extension Division, Berkeley. It consisted of ten sessions for which 
three quarter units of credit were awarded. The course materials used 
at this site were essentially the same as those tested in the main field 
test. 

Since the chief objective of this field test was to establish 
whether the Instructor materials and guidelines were sufficiently 
developed for independent instructor use. Dr. Mattas received the 
materials with minimal consultation or preplanning, which required him 
to rely on his own judgment to organize and conduct the training and 
obtain evaluation data. 

2. Resul ts 

As stated above, there were three objectives for this pre- 
operational field test. Careful review of the information obtained from 





49 



trainees, the instructor and the observer indicate that these three 
objectives were met with varying degrees of success. 

The positive value of the training for personnel already working in 
an information dissemination role was demonstrated in three ways: (a) 
through analysis of the course pretest/posttest data; (b) from a report 
submitted by the Center Director (Dr. Mattas) attesting to positive 
changes in the behavior of the members of his staff who completed the 
EIC training; and (c) from statements made by these same members six 
weeks later regarding the perceived value of the course to them. 

Appendix A-2 presents the overall course exercise used in the 
evaluation of the San Mateo site test (Site 3). Table 6 presents the 
subjectively scored pretest/posttest results for the five trainees who 
completed both exercises. This table indicates the number who exhibited 
an increase, no change, and a decrease, as rated by two independent 
judges. 

Test Items 1 through 5 present the trainee with a simulated request 
from a client and a set of possible alternative responses that an EIC 
could make. The trainee is requested to select his "best" response and 
to provide a rationale for his selection. The rationale provided by the 
trainee is rated according to a five point scale, with "5" being "excellent" 
and "1" being !: .poor." 

Table 6 indicates that a majority of trainees exhibited an increase 
in their ratings from pretest to posttest on each of these items. 
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Results 


TABLE 6 

of the Pretest/Posttest Comparisons of Subjectively 


Scored Items in the Overall Course Diagnostic Exercise for Site #3 




Item #1 


Item #2 


Item #3 


Item #4 


Item #5 


Rater 


N M 


N M 


N M 


N M 


N M 


Increase 


4 5 


3 3 


4 3 


3 4 


4 2 


No Change 


0 0 


2 2 


0 1 


0 1 


0 2 


Decrease 


1 0 


0 0 


1 1 


2 0 


1 1 


Percent 


80% 


60% 


40% 


60% 


60% 


Agreement 


Item #6 


Item #7 


Item #8 


Item #9 


Item #10 


Rater 


N M 


N M 


N M 


N M 


N M 


Increase 


4 4 


1 2 


4 4 


5 4 


3 4 


No Change 


0 0 


3 2 


0 0 


0 1 


2 1 


Decrease 


1 1 


1 1 


1 1 


0 0 


0 0 


Percent 


100% 


' 80% 


100% 


80% 


80% 


Agreement 













Items 6 through 9 are factual or informational items related to the 
simulated client's request. Table 6 indicates that Items 6, 7, and 8 
reflected an increase in rating for four out of the five trainees, whereas 
Item 7 reflected a gain in only one trainee.- as judged by Rater N and in 
two trainees as judged by Rater M. This item required the trainee to list 
at least five primary sources to be used in beginning a search. The lack 
of improvement from pretest to posttest on this item was probably due to 
the use of the term "primary" which, during the course, took on specific 
technical meaning. Item 10 in Table 6 is a comprehensive rating based on 
the extent to which the trainee mad*? use of appropriate terminology in 
his answers to Items 1 through 9. As indicated in Table 6, the majority 
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of trainees exhibited an increase in their rating on this item. Table 
6 also presents the percentage of agreement between the two raters for 
each item. As expected, the agreement on the informational items (6 
through 9) is higher than on items requiring a more subjective evaluation 
(1 through 5). The agreement between the two independent raters across 
all ten items was 74%. 

Table 7 presents the results of the objectively scored items in the 
overall course exercise for Site #3. This table indicates the number of 
trainees who exhibited an increase, no change, and a decrease cn each of 
these four items. The findings indicate that a majority of trainees 
increased on each of these knowledge items. 







TABLE 7 






Results of the 


Pretest/Posttest Comparisons 


of Objectively 


Scored Items in the 


Overall 


Course Diagnostic Exercise for Site #3 


Item No. 


a 


b 


c 


d 


Increase 


3 


4 


4 


3 


No Change 


0 


0 


0 


2 


Decrease 


2 


1 


1 


0 



The report submitted by the instructor six weeks after completion 

of the course included the following observations: 

I have noted two major positive changes in the five 
members of my staff who took the course. They are 
more aware of the variety of services, indexes, 
compendia, "handbooks, etc. available for their use 
and are more willing to expand their search efforts 
to the full range of possible resources. Secondly, 
they have become sensitive to the large number of 
personal judgments required in this role, and have 
become aware of the influence of their own biases 
in the retrieval process and the effect of their 
behavior in the communication process. As a result 
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